Positioning a label using a virtual label and real-time image of a target field of view

ABSTRACT

The disclosure discloses a non-transitory computer-readable recording medium storing a virtual label display process program for executing steps. The steps include a composite image generating step, a composite image output step, a determining step, and a notifying step. In the composite image generating step, a real image data of a desired field of view and a virtual image data of a label are combined. In the composite image output step, a composite image data is output to a display device, and a virtual image of the label on the display device is superimposed and displayed. In the determining step, it is determined whether a desired suitability is satisfied between an exterior appearance of a background object and an exterior appearance of the label based on the real image data and the virtual image data. In the notifying step, a predetermined suitability notification is made.

CROSS-REFERENCE TO RELATED APPLICATION

This application is a continuation application of U.S. application Ser.No. 15/714,127 filed on Sep. 25, 2017, which claims benefit of JapanesePatent Application No. 2016-190335, which was filed on Sep. 28, 2016,the disclosure of which is incorporated herein by reference in itsentirety.

BACKGROUND Field

The present disclosure relates to a recording medium storing a virtuallabel display process program executed by an operation terminaloperating a label producing device.

Description of the Related Art

Techniques of producing a label pasted and used on an object to beaffixed through operation of a label producing device from an operationterminal are well known. In this kind of techniques, a print image of alabel to be produced is displayed on a display monitor included in anoperation terminal such as a personal computer (PC). When an operatorperforms an operation to input characters, graphics, etc. while viewingthe image display, the corresponding print data is generated in theoperation terminal and transmitted to the label producing device. As aresult, a print corresponding to the print data is performed on atransported tape by a thermal head in the label producing device, and alabel is produced. In another known technique, a printed image to beposted suitable for a positing place is easily produced based onenvironmental information in a place where a label is planned to beposted.

In the prior arts described above, when a label is actually produced, anoperator judges the suitability to the environment in which the label isplanned to be pasted. Therefore, it is disclosed that “the place topaste the label is already decided and printing contents (template)suitable to the place are determined”. However, print data may beproduced in advance when a multiplicity of objects to be affixed exists,for example, as in the case of fixtures in a workplace, personalbelongings brought to a school, etc. In this case, since the print datais produced before determining where a label is pasted, the suitabilitycannot be judged between the produced label and a place to paste thelabel. Therefore, needs for simulation cannot be met when “a label(including printing contents) is already decided and a label pastingplace suitable for the label is determined”.

SUMMARY

An object of the present disclosure is to provide a recording mediumstoring a virtual label display process program capable of finding amost suitable place to paste a label by using print data for a labelproduced in advance and capable of improving convenience for a user.

In order to achieve the above-described object, according to the aspectof the present application, there is provided a non-transitorycomputer-readable recording medium capable of reading by a computingdevice, storing a virtual label display process program for executingsteps on the computing device, the computing device provided in aportable terminal that is configured to operate a label producing deviceconfigured to produce an attachable label and includes an operationdevice, an imaging device, a display device, and the computing device,the steps comprising a virtual image generating step for generatingvirtual image data of the label, a real image acquiring step foracquiring real image data of a desired field of view photographed by theimaging device, a composite image generating step for combining the realimage data acquired in the real image acquiring step and the virtualimage data generated in the virtual image generating step to generatecomposite image data, a composite image output step for outputting thecomposite image data generated in the composite image generating step tothe display device, and superimposing and displaying a virtual image ofthe label in an image of the desired field of view on the displaydevice, a determining step for determining whether a desired suitabilityis satisfied between an exterior appearance of a background objectincluded in the desired field of view and an exterior appearance of thelabel based on the real image data acquired in the real image acquiringstep and the virtual image data generated in the virtual imagegenerating step, and a notifying step for making a predeterminedsuitability notification in the case that it is determined that thesuitability is satisfied in the determining step.

The virtual label display process program of the present disclosure isused for finding a most suitable place to paste a label to be producedby a user. In particular, when the virtual label display process programis executed by the computing device of the portable terminal, thevirtual image data is first generated (virtual image generating step).This virtual image data may reflect a result of an editing operatorperformed on an appropriate editing screen through the operation deviceby the user or may be the print data of the label completed in advance,for example. When the user then photographs a desired field of viewincluding an object to be affixed with the imaging device of theportable terminal, the real image data corresponding to the photographedresult is acquired (real image acquiring step). As a result, thegenerated virtual image data and the acquired real image data are thencombined (composite image generating step) and the composite image datais output to the display device (composite image output step).Therefore, the virtual image of the label is displayed in an overlappedstate in the image of the desired field of view on the display device,so that the user can easily have a visual image of a virtual appearanceform when the label is attached to the object to be affixed in the imageof the field of view.

The portable terminal of the present disclosure can determine thedesired suitability between the exterior appearance of the object to beaffixed and the exterior appearance of the label pasted thereon. It isconceivable that examples of the suitability include whether the labellooks clearly floating up when the label is attached to the object to beaffixed (a degree of clarity as a feeling of sharpness) or, conversely,whether the label looks moderately buried when the label is attached tothe object to be affixed (a degree of being buried as a feeling ofinconspicuousness), etc. In the present disclosure, based on the realimage data and the virtual image data, it is determined at thedetermining step whether the desired suitability is satisfied. If thedetermination is affirmative, suitability notification is made at thenotifying step to notify the user that the suitability is satisfied.

As a result of the process as described above, the user can move theportable terminal or move around with the portable terminal held by handto take images of various places while displaying the virtual image ofthe label to be produced displayed on the display device, so as to findthe most suitable place to paste the label. Consequently, theconvenience for the user can be improved.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1A is a front view showing an exterior appearance of an operationterminal executing a virtual label display process program.

FIG. 1B is a back view showing the exterior appearance of the operationterminal executing the virtual label display process program.

FIG. 2 is a front view showing an exterior appearance of a labelproducing device that is an operation object of the operation terminal.

FIG. 3 is a functional block diagram showing functional configurationsof the operation terminal and the label producing device.

FIG. 4 is a flow showing control procedures related to label productionexecuted by a control circuit of the label producing device.

FIG. 5 is a front view showing an example of an exterior shape of aproduced label.

FIG. 6 is an explanatory view showing a layout screen of print data of alabel.

FIG. 7 is an explanatory view showing a virtual label display screen.

FIG. 8A is an explanatory view of an imaging state at the time ofcomposite display of a virtual label image and an object real image.

FIG. 8B is an explanatory view of an affixing position setting(searching) screen in which an imaged object real image and a virtuallabel image are compositely displayed at the time of composite displayof the virtual label image and the object real image.

FIG. 9 is a flowchart showing control procedures executed by a controlcircuit of the operation terminal.

FIG. 10 is a front view showing an exterior appearance of another labelproducing device that is an operation object of the operation terminal.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

An embodiment of the present disclosure will now be described withreference to the drawings.

In FIGS. 1A and 1B, a portable terminal 10 has a device main body 11 andan imaging part 12 having a camera function.

The device main body 11 includes a display part 13 made up of, forexample, a touch-panel type color liquid crystal display, displayingvarious pieces of information and various messages, and allowing anoperator to perform various operations, and an operation part 14 such asan operation button that the operator can operate.

The imaging part 12 is disposed to an upper portion (upper portions ofFIGS. 1A and 1B) of the device main body 11 and includes a camera lens12A on the side opposite to a surface on which a display surface of thedisplay part 13 is disposed, i.e., on the back side. For example, whenthe operator operates the operation part 14 to display an appropriateicon etc. on the display part 13 and selects the icon by pressing theicon with a finger, the imaging part 12 is activated by a touch panelfunction, and an object can be photographed (a moving image or a stillimage can selectively be photographed).

In FIG. 2, a label producing device 20 has a device main body 21. Thedevice main body 21 includes, for example, a display part 22 made up ofa liquid crystal display and displaying various pieces of informationand various messages, an operation part 23 having a plurality of buttonswitches etc. allowing an operator to perform various operations, and alever 24 for driving a cutter 31 (see FIG. 3 described later) disposedinside the device main body 21.

The operation part 23 includes various function keys 23 a includinginput keys for inputting characters, symbols, numbers, etc., movementkeys 23 b functioning as movement keys for a cursor position at the timeof character input and transition keys at the time of mode selection,etc. The operation part 23 can allow one key to provide a plurality offunctions.

In FIG. 3, the portable terminal 10 includes, in addition to the imagingpart 12, the display part 13, and the operation part 14 described above,a control circuit part 15, a storage part 16 for storing various piecesof information, a ranging part 17 for distance measurement etc. usablefor focusing etc., when using the camera function, and a communicationcontrol part 18.

The control circuit part 15 includes a CPU, a RAM, and a ROM not shown.This control circuit part 15 executes various programs stored in advancein the ROM while using a temporary storage function of the RAM by anarithmetic function of the CPU. These various programs include a virtuallabel display process program of the present disclosure. The controlcircuit part 15 is connected via the communication control part 18 tothe label producing device 20 through a wireless communication and cantransmit/receive various pieces of information and various data to/fromthe label producing device 20.

The storage part 16 is made up of, for example, a nonvolatile memorysuch as an electrically erasable programmable read-only memory (EEPROM)in which storage contents can be added and erased. This storage part 16stores various programs for production of a label L and suitabilitydetermination (see a flow described later) using the camera function, inaddition to the virtual label display process program related to thepresent disclosure executed by (the CPU of) the control circuit part 15.

The communication control part 18 controls information communicationperformed with the label producing device 20 connected wirelessly.

In FIG. 3, the label producing device 20 includes, in addition to thedisplay part 22, the operation part 23, the lever 24, and the cutter 31described above, a control circuit part 25, a storage part 26 storingvarious pieces of information, a communication control part 27, acartridge holder 32, a feeding device 33, and a printing head 34.

The control circuit part 25 includes a CPU, a RAM, and a ROM not shown.This control circuit part 25 executes various programs stored in advancein the ROM while using a temporary storage function of the RAM. As aresult, the control circuit part 25 generally controls the labelproducing device 20. The control circuit part 25 is connected to theportable terminal 10 via the communication control part 27 and cantransmit/receive various data to/from the portable terminal 10.

The storage part 26 is made up of, for example, a nonvolatile memorysuch as an electrically erasable programmable read-only memory (EEPROM)in which storage contents can be added and erased. The storage part 26can store print data transmitted from the portable terminal 10.

The communication control part 27 controls information communicationperformed with the portable terminal 10 connected wirelessly asdescribed above.

The cartridge holder 32 is configured to attach and detach a cartridge36 capable of supplying a print-receiving tape 35. The cartridge 36includes a tape roll 37 (shown as concentric circles in a simplifiedmanner although actually having a spiral shape) having theprint-receiving tape 35 wound around a reel member not shown. Thecartridge holder 32 acquires tape color information (tape color and tapewidth) of the set cartridge 36 and transmits the information to thecontrol circuit part 25.

The feeding device 33 is disposed facing the printing head 34. Thisfeeding device 33 is controlled by the control circuit part 25 totransport the print-receiving tape 35 supplied from the cartridge 36mounted on the cartridge holder 32, i.e., fed out from the tape roll 37of the cartridge 36.

The printing head 34 is controlled by the control circuit part 25 toperform a desired print on the print-receiving tape 35 fed out from thetape roll 37 and transported by the feeding device 33.

When the lever 24 is operated by the operator, the cutter 31 cuts theprint-receiving tape 35 after completion of the print by the printinghead 34 in a predetermined length to form one printed label L (see FIG.5 described later).

Procedures of producing the label L by the label producing device 20will be described with reference to FIG. 4.

In FIG. 4, a process shown in this flow starts when the portableterminal 10 outputs a label production instruction to the labelproducing device 20 (“START” position). First, at step S1, the controlcircuit part 25 outputs a control signal to the feeding device 33 tostart feeding-out of the print-receiving tape 35 from the tape roll 37of the cartridge 36 mounted on the cartridge holder 32, i.e.,transporting of the print-receiving tape 35.

The flow goes to step S2, and the control circuit part 25 determineswhether a predetermined amount of the print-receiving tape 35 istransported. The predetermined amount is, for example, a transportdistance (a so-called transport amount for a front margin) allowing aleading end of a print area not shown of the print-receiving tape 35 toreach a position substantially facing the printing head 34. Thedetermination of step S2 is negative (step S2:NO) during waiting in aloop until the predetermined amount is transported and, when thepredetermined amount is transported, the determination of step S2becomes affirmative (step S2:YES), and the flow goes to step S3.

At step S3, the control circuit part 25 outputs a control signal to theprinting head 34 to start a print of print data stored in the storagepart 26 onto the print area of the print-receiving tape 35 transportedby the feeding device 33.

Subsequently, at step S4, the control circuit part 25 determines whetherall the print of the print data onto the print area of theprinting-receiving tape 35 by the printing head 34 started at step S3 iscompleted. The determination of step S4 is negative (step S4:NO) duringwaiting in a loop until all the print is completed and, when all theprint is completed, the determination of step S4 becomes affirmative(step S4:YES), and the flow goes to step S5.

At step S5, the control circuit part 25 determines whether apredetermined amount of the print-receiving tape 35 (e.g., a transportdistance allowing all the print area to exceed the cutter 31 by apredetermined length. a so-called transport amount for a rear margin) isfurther transported. The determination of step S5 is negative (stepS5:NO) during waiting in a loop until the predetermined amount istransported and, when the predetermined amount is transported, thedetermination of step S5 becomes affirmative (step S5:YES), and the flowgoes to step S6.

At step S6, the control circuit part 25 outputs a control signal to thefeeding device 33 to stop the feeding-out of the print-receiving tape 35from the tape roll 37 started at step S2, i.e., the transporting of theprint-receiving tape 35.

At step S7, the control circuit part 25 outputs a display signal to thedisplay part 22 to perform display indicating that the print-receivingtape 35 can be cut by operating the lever 24. Subsequently, the processshown in this flow is terminated. When the operator operates the lever24 in accordance with the display, the cutter 31 is actuated and theprint-receiving tape 35 is cut. Because of this cutting by the cutter31, a portion printed by the printing head 34 is separated from theprint-receiving tape 35 as the one label L.

As shown in FIG. 5, the label L produced as described above is arectangular label having a width Lw and a length Lg. In this example, aprint RR (character string “UENO-TECH-10450”) corresponding to the printdata is printed on a surface of the label L.

In the basic configuration described above, the present embodiment ischaracterized in that before the label producing device 20 actuallyproduces the label L as shown in FIG. 5, the portable terminal 10displays a virtual label appearance equivalent to the label L(hereinafter appropriately referred to as a “virtual label”) togetherwith an image of a background object photographed by the imaging part 12so as to allow the portable terminal 10 to determine the suitabilityregarding whether it is preferable to actually paste the label L on thebackground object. The details will hereinafter be described in order.

<Transition in Display State of Screen>

First, the operator performs appropriate operational input through thedisplay part 13 and the operation part 14 to display a “layout” screenshown in FIG. 6 on a screen of the display part 13. This “layout” screenon the screen of the display part 13 has a work content display field 13a displaying characters “layout”, a print data entry field 13 bimitating a print tape and displaying print data R such as inputcharacters, and an operation button field 13 c displaying a “producelabel” button and a “virtual affixing” button operated as switchesthrough the touch panel function.

While this “layout” screen is displayed, the operator inputs desiredtext characters as shown in FIG. 6 (the character string“UENO-TECH-10450 . . . ” in the shown example. a cursor C is displayedat the end) by an operation such as tracing and writing characters witha finger by using the touch panel function inside (or, conversely,outside) of the print data entry field 13 b so as to set the print dataR.

At this time, the length of the label L is set by the control circuitpart 15 in accordance with the contents (e.g., the number of characters)of this print data R. The width of the label L is fixedly defined inadvance or is set by the same operator's operation as described above inan entry field (or a different screen, window, etc.) not shown. In thisexample, the length dimension and the width dimension of the label L are35 mm and 9.5 mm, respectively, which are displayed on the screen of thedisplay part 13 outside and near the print data entry field 13 b. Thecontents of the print data R of the label L set by such an editingoperation and the length dimension/width dimension of the label L arestored in the storage part 16.

After completion of the editing operation, the operator presses(touches) the “produce label” button in the operation button field 13 cto transmit the print data R generated as described above via thecommunication control part 18 to the label producing device 20. As aresult, the flow shown in FIG. 4 is executed, and the label producingdevice 20 produces the label L on which the print RR corresponding tothe print data R is printed as shown in FIG. 5.

On the other hand, after completion of the editing operation, theoperator presses the “virtual affixing” button in the operation buttonfield 13 c. As a result, the screen of the display part 13 is switchedto a “virtual label display” screen shown in FIG. 7. This “virtual labeldisplay” screen on the screen of the display part 13 has the workcontent display field 13 a displaying characters “virtual labeldisplay”, a virtual label LI displayed thereon, and the operation buttonfield 13 c displaying a video icon operated as a switch through thetouch panel function. The video icon is pressed (touched) at thestart/end of photographing of a moving image. In this case, a cameraicon for photographing a still image may be displayed instead of thisvideo icon, or the video icon and the camera icon may be displayed.

The switched “virtual label display” screen displays the virtual labelLI reflecting the editing operation on the “layout” screen of FIG. 6,i.e., an image of the label L to be actually produced, in a desired sizein a central part on the screen of the display part 13. In this case,the display position of the virtual label LI is fixed on the “virtuallabel display” screen. The operator then utilizes the camera function ofthe portable terminal 10 to locate an intended object to be affixed towhich the actual label L is attached, i.e., a background object forwhich the suitability is determined (e.g., an electric pot P describedlater), at a position falling within the field of view of the imagingpart 12 and presses the video icon in the operation button field 13 c.As a result, the imaging part 12 starts photographing a moving image ofthe object to be affixed (the background object. the same applieshereinafter).

FIGS. 8A and 8B show an example of a display state of the display part13 after the photographing of the moving image is started in this way.For example, when the object to be affixed (e.g., the electric pot P) isphotographed as an object by the imaging part 12 as shown in FIG. 8A, an“affixing position setting (searching)” screen is displayed as shown inFIG. 8B such that a photographed real image (hereinafter appropriatelysimply referred to as an “object image PI”) is overlapped with thevirtual label LI as the background of the virtual label LI. Duringphotographing, a “REC” mark is displayed in the vicinity of the videoicon in the operation button field 13 c for notification of recording inprogress. The notification may be made in any form as long as it can berecognized that recording is in progress.

When the operator holding the portable terminal 10 by hand changes thepositional relationship with the object to be affixed, the position andthe size of the object image PI displayed on the screen of the displaypart 13 also change in real time; however, the virtual label LI iscontinuously fixedly displayed and maintained in the desired size in thecentral part. This enables the control circuit part 15 of the portableterminal 10 to determine the suitability in consideration of therelative positional relationship between the object image PI and thevirtual label LI, in terms of the suitability that “the produced label Lis preferably (conspicuously/inconspicuously) attached to this place ofthe object”, by replacing the virtual label LI on the object image PIthrough virtual affixing.

In this case, the screen of the display part 13 displays a plurality ofcolor selection icons AC each having a pair of a tape color and acharacter color such that the color of characters and the tape color ofthe generated virtual label LI can be changed. The color selection iconsAC are pairs of preset character colors and tape colors. The colorselection icons AC are arranged such that the operator can make aselection by appearance. For example, the colors of the tape color onthe left side and the colors of the character color on the right sideare displayed in a list format on the display part 13. Icon informationfor displaying these color selection icons AC on the display part 13 canbe included in an application. In this case, the portable terminal 10may acquire the information from the label producing device 20 as a partof update information of the application, for example. Alternatively,for example, the portable terminal 10 may cause the label producingdevice 20 to identify through the cartridge holder 32 the type of thecartridge 36 set in the cartridge holder 32, before acquiring thecurrent tape color information (the tape color and the tape width). As aresult, the portable terminal 10 can make the icon display correspondingto the color of the currently mounted tape different from (e.g., largerthan) the other icon display out of the color selection icons AC on thedisplay part 13 so that the operator can recognize the display. In thisway, for example, the control circuit part 15 of the portable terminal10 determines the suitability in terms of whether it is preferable toattach the label L to the object based on compatibility of the tapecolor and the character color of the virtual label LI with the color ofthe photographed object image PI etc., and makes a notification at leastwhen a preferably determination result is acquired (details will bedescribed later).

In such a basic configuration, when the label L to be printed iscompleted as the print data R, and the contents such as characters andgraphics to be printed and the shape/exterior appearance such as thecolor and the size of the print-receiving tape 35 are already decided,the operator can move while photographing with the portable terminal 10such as a camera-equipped smartphone and display the virtual image ofthe label L (the virtual label LI) superimposed on the photographedbackground image in real time, so as to perform a simulation forsearching around for a most suitable position to paste the label L. As aresult, when the control circuit part 15 determines that the suitabilityis high (or determines a degree of the suitability) by comparing thecolor information of the background image changing every moment with thecolor information of the label, this determination is displayed as amessage M or the operator is notified by sound or vibration, so that themost suitable place to paste the label can be found, and the conveniencefor a user can be improved.

For example, the control circuit part 15 has sample data indicative ofthe suitability of color coordination stored in a table format etc. inthe storage part 16 and can utilize the stored sample data to determinethe suitability between the virtual label LI and the object image PI.The hue of the object can be recognized by a two-dimensional colorcharge-coupled device (CCD) disposed on the imaging part 12.

<Details of Control>

The details of control performed by the control circuit part 15(specifically, the CPU) so as to implement the display contents(simulation) as described above will be described with reference to aflowchart of FIG. 9.

In FIG. 9, first, at step S11, the control circuit part 15 outputs adisplay signal to the display part 13 to display the “layout” screen asa screen for operation and editing (see FIG. 6) and then acquires theprint data R in accordance with contents of operational input by theoperator for producing or editing the label L. Subsequently, the flowgoes to step S12 in response to the operation of the “virtual affixing”button described above. If the operator does not produce or edit thelabel L, i.e., if the already produced print data R is used, the printdata R may simply be acquired from the storage part 16. In such a casethat the print RR etc. of the label L are fixedly defined (by using atemplate etc.) or that the production/editing is performed by usinganother device, the routine of the control circuit part 15 related tothe production/editing operation at step S11 is eliminated and, forexample, the print data R including the current tape color informationis acquired.

At step S12, the control circuit part 15 generates virtual image datafor displaying the virtual label LI corresponding to the image of thelabel L on the screen of the display part 13 based on the print data Rincluding the current tape color information acquired at step S11, andgoes to step S13. The generated virtual image data is output to thedisplay part 13 and the virtual label LI is displayed on the screen ofthe display part 13 (see FIG. 7).

At step S13, the control circuit part 15 acquires real image data(including image data of the object image PI) of a desired field of viewincluding the object (electric pot P) photographed by the operator withthe imaging part 12. Subsequently, the flow goes to step S14.

At step S14, the control circuit part 15 combines the virtual image datagenerated at step S12 and the real image data acquired at step S13 togenerate composite image data. Subsequently, the flow goes to step S15.

At step S15, the control circuit part 15 outputs the composite imagedata generated at step S14 to the display part 13 for superimposing anddisplaying the virtual label LI and the object image PI on the screen ofthe display part 13 (see FIGS. 8A and 8B).

The flow goes to step S16 and the control circuit part 15 acceptssetting of a mode at the time of causing the control circuit part 15 todetermine the suitability through the operator's operation of theportable terminal 10 and uses the setting as at least a portion ofconditions for making the suitability determination in accordance withthe accepted mode.

For example, if the control circuit part 15 includes a known imagerecognition process function, the real image data acquired at step S13is analyzed to extract at least two points having features different theother positions, such as a contour, an edge portion, recess andprotrusion portions, a corner portion, and a black portion, so as to setfeature points. In this example, two points of a handle portion of theelectric pot P are set as feature points in the object image PI (seeFIGS. 8A and 8B). These feature points may also be set arbitrarily todesired points by the operator through an appropriately operation on thescreen of the display part 13 in the same way as described above.

The flow goes to step S17 and, on the basis of the conditionscorresponding to the mode set at step S16, the control circuit part 15determines whether the desired suitability is satisfied between theexterior appearance of the object to be affixed and the exteriorappearance of the label L based on the virtual image data of the virtuallabel LI generated at step S12 and the real image data acquired at stepS13. If determining that the desired suitability is satisfied, thecontrol circuit part 15 goes to step S18. On the other hand, ifdetermining that the desired suitability is not satisfied, the controlcircuit part 15 repeats this routine until determining that thesuitability is satisfied. However, the control circuit part 15terminates this routine if the operator performs an operation offorcibly terminating this suitability determination or if it isdetermined a predetermined number of times or more that the suitabilityis not satisfied. If a moving image is photographed, the determinationon whether the desired suitability is satisfied is triggered by apredetermined determination start operation by the operator. In thisregard, if a moving image is photographed, the determination of thesuitability may constantly be made in real time. In this case, thecontrol circuit part 15 extracts the color information of the real imagedata and the color information including the label of the print data andcalculates and compares the suitability of these two pieces of the colorinformation every moment (whether the label L is conspicuous against abackground wall or object in a mode in which such a combination isdesired or whether the label L is buried in a background wall or objectin a mode in which such a combination is desired), so that thedetermination can be made.

The flow goes to step S18, and the control circuit part 15 notifies theoperator that the desired suitability is satisfied, and terminates thisroutine. A technique of the notification may include displaying themessage M of the notification on the display screen of the display part13, sounding of a voice, etc. Specifically, when a determination resultrelated to the suitability reaches a certain threshold value or more, itis considered that “this is the best place to paste this label!”, and anotification is immediately made at this time point by a display of“this is the best place for your label” etc. on a smartphone screen, ablinking of the label virtual image, a voice, or vibrations. It is notedthat even if it is determined at step S17 that the suitability is notsatisfied, a notification can successively be made by a message etc. toindicate that the label L is not suitable in this place.

This allows the operator to recognize the optimal place to paste thelabel. In this case, the determination of the suitability between thevirtual label LI and the object may include a color change associatedwith light and shade as the color of the background, i.e., the object tobe affixed. For the color of the label L, a color of ink used for printformation on the label L may be taken into consideration in addition tothe color of the print-receiving tape 35. The color information of thedetermination method is calculated by using RGB values. In this case,the determination of the suitability may be made in accordance not onlywith a simple hue but also with a hue corresponding to a time zone,season, etc. As a result, for example, a hue giving a positive feelingdiffers even between a label of a dish served in a breakfast buffet of ahotel and a label of the same dish served at night. Therefore, such acondition may be included in criteria for determination.

Furthermore, it is assumed that the determination of the suitability ismade in accordance with the use purpose of the label L, for example,when it is desired to make the label conspicuous against the background,and conversely, when it is desired to bury the label in the backgroundas inconspicuous as possible. For example, when the print contents arecharacters such as “don't forget to take your umbrella with you!” andthe color of the tape is blue, the object suitable for the blue can besearched for to determine the suitability including not only the placebut also an image corresponding to the purpose.

When it is desired to make the label conspicuous against the background,the suitability can be determined based on whether a difference betweenthe background color (RGB value) and the tape color (RGB value) is equalto or greater than a predetermined threshold value set considerablylarge. Conversely, when it is desired to bury the label L in thebackground as inconspicuous as possible, the suitability can bedetermined based on whether a difference between the background color(RGB value) and the tape color (RGB value) is within a predeterminedrange (e.g., equal to or greater than a predetermined threshold valueclose to zero and equal to or less than a significantly small thresholdvalue larger than the predetermined threshold value). It is noted thatthe determination may be made based on the combination of colorcoordination described above. Therefore, such a suitabilitydetermination can be applied to the already produced print data R ingeneral, i.e., not only to “searching for the background suitable forthe label L” but also to a technique of “searching for the labelsuitable for the background”.

Furthermore, relative sizes of the object to be affixed and the label Lcan also be used as a factor in the determination of the suitability interms of conspicuousness and inconspicuousness. In particular, even thelabel L of the same size becomes conspicuous when pasted in a narrowspace and becomes less conspicuous when pasted in a large space.Therefore, such an area ratio between the object and the label L etc.can be included in the determination conditions.

The ranging part 17 measures a distance between the portable terminal 10and the object by a known distance measurement technique such as a lightsensor, for example, and acquires from the angle of view the actual sizeof the object, such as external dimensions and recess and protrusionshapes. Therefore, the control circuit part 15 may display the virtuallabel LI at a magnification ratio changed in accordance with theacquired external dimensions etc. of the object.

As described above, the portable terminal 10 executes step S12 ofgenerating virtual image data, step S13 of acquiring real image data ofa desired field of view photographed by the imaging part 12, step S14 ofcombining the acquired real image data and the generated virtual imagedata to generate composite image data, step S15 of outputting thegenerated composite image data to the display part 13 for superimposingand displaying a virtual label LI in the image of the desired field ofview on the display part 13, step S17 of determining whether a desiredsuitability is satisfied between an exterior appearance of a backgroundobject included in the desired field of view and the exterior appearanceof the label L based on the acquired real image data and the generatedvirtual image data, and step S18 of making a predetermined suitabilitynotification (display on the display part 13; voice notification;vibration notification) if it is determined that the suitability issatisfied.

Therefore, the virtual label display process program is used for findinga most suitable place to paste the label L to be produced by theoperator. In particular, when the virtual label display process programis executed by the control circuit part 15 of the portable terminal 10,the virtual image data is first generated. This data may reflect aresult of an editing operator performed on an appropriate editing screenthrough the touch panel of the display part 13 by the operator or may bethe print data of the label L completed in advance, for example. Whenthe operator then photographs a desired field of view with the imagingpart 12 of the portable terminal 10, the real image data correspondingto the photographed result is acquired. As a result, the virtual imagedata and the real image data are combined and the composite image datais output to the display part 13. Therefore, the virtual label LI isdisplayed in an overlapped state in the image of the desired field ofview on the display part 13, so that the operator can easily have avisual image of a virtual appearance form when the label L is attachedto the object to be affixed (background object) in the image of thefield of view.

As described above, the portable terminal 10 can determine the desiredsuitability between the exterior appearance of the object to be affixedand the exterior appearance of the label L pasted thereon. It isconceivable that examples of the suitability include whether the label Llooks clearly floating up when the label L is attached to the object tobe affixed (an increased degree of clarity as a feeling of sharpness)or, conversely, whether the label L looks moderately buried when thelabel L is attached to the object to be affixed (a degree of beingburied as a feeling of inconspicuousness, in other words, a decreaseddegree of clarity), etc. Based on the real image data and the virtualimage data, it is determined whether the desired suitability issatisfied. If the determination is affirmative, a suitabilitynotification is made to notify the operator that the suitability issatisfied.

As a result of the process as described above, the operator can move theportable terminal 10 or move around with the portable terminal 10 heldby hand to take images of various places while displaying the virtuallabel LI to be produced displayed on the display part 13, so as to findthe most suitable place to paste the label L. Consequently, theconvenience for the operator can be improved.

In this case, by generating the composite image data in which thedisplay position of the virtual label LI on the display part 13 issubstantially fixed regardless of variation in the field of view, thevirtual label LI can be displayed at the same position in the displaypart 13 without change even when the operator moves the portableterminal 10 or moves around with the portable terminal 10 held by handto take images so that the field of view displayed on the display part13 changes every moment.

As indicted by three black dots in FIG. 8B, a triangular positionalcorrelation can be set between the object image PI and the virtual labelLI so as to maintain a certain fixed correspondence relationship. Inthis example, the virtual label LI has the upper left corner of thevirtual label LI set in advance as the reference point of the positionalcorrelation (see FIG. 8B). This reference point may be set to anarbitrary position of the virtual label LI through an appropriateoperation on the portable terminal 10 by the operator. In the objectimage PI, the two points of the handle portion of the electric pot Pserving as the feature points described above are set as the referencepoints of the positional correlation (see FIG. 8B). Therefore, even ifthe object image PI displayed on the display part 13 moves or changes insize due to variation in the field of view imaged by the imaging part12, the virtual label LI can be moved or changed in size so as to followthe movement and the change. Consequently, even though the label L isnot yet attached to the object to be affixed (the electric pot P in thisexample) in the real world, the display on the display part 13 canvirtually achieve the state as if the label L is actually attached tothe object. As a result, the form after the attachment can visually besimulated before actually attaching the label L to the object.Additionally, since the virtual label LI can be simulated in variousstates with different fields of view, the operator can acquire a feelingcloser to the actually attached state, and the convenience can beimproved.

Additionally, since the suitability is determined based on the colorinformation of a predetermined range (affixing position) narrower thanthe desired field of view and including the object to be affixed in thereal image data and the color information of the virtual image data, thedetermination of the suitability can be performed through a numericalarithmetic process by using RGB values etc. as the color information,for example.

By determining as the desired suitability at least whether an increaseddegree of clarity (a feeling of sharpness) of the label L is equal to orgreater than a predetermined threshold value when the label L isattached to the object to be affixed, the optimal place can be foundsuch that the label L to be produced by the operator looks clearest whenpasted.

Conversely, by determining as the desired suitability at least whether adegree of being buried, i.e., a decreased degree of clarity (a feelingof inconspicuousness), of the label L is within a predetermined rangewhen the label L is attached to the object to be affixed, the optimalplace can be found such that the label L to be produced by the operatorlooks moderately buried when pasted.

In this way, by causing the control circuit part 15 to further executestep S16 of accepting selection input of contents of the desiredsuitability via the touch panel of the display part 13 so as todetermine whether the accepted desired suitability is satisfied betweenthe exterior appearance of the object to be affixed and the exteriorappearance of the label L, the operator can selectively input, forexample, whether to search for the place in which the label L to beproduced looks clearest when pasted, or whether to search for the placein which the label L to be produced looks moderately buried when pasted.

The control circuit part 15 generates the composite image data on thebasis of the exterior appearance of the object to be affixed based onthe real image data by combining and superimposing the virtual imagedata in a form having the exterior appearance size of the label Lchanged in magnification ratio with the real image data and, therefore,for example, when the label L should better be changed in magnificationratio (enlarged or reduced) so as to satisfy the desired suitability(desired to make the label conspicuous or moderately buried) because ofthe color and the shape of the object to be affixed, the control circuitpart 15 can make the operator visually recognize this fact.

Moreover, by causing the control circuit part 15 to further execute stepS11 of accepting an editing operation to the print contents of the labelL so as to generate virtual image data reflecting the accepted editingoperation, the virtual label LI can be displayed in the display part 13in the form reflecting the result of the editing operation performed bythe operator for the print displayed on the label L.

Particularly, in this embodiment, the operator can perform the editingoperation on the layout screen (see FIG. 6) displayed on the displaypart 13 so as to set the form of the label L such as the print RR etc.as desired. As a result, the virtual label LI corresponding to the labelL having the print RR etc. set as desired by the operator in this waycan be displayed in the state of being virtually attached to the object(before actually producing the label L).

In the example used in the above description, the object to be affixed(the electric pot P in this example) is photographed by the portableterminal 10 while taking an image with the imaging part 12, so as togenerate the virtual label LI based on the print data R for the label Lproduced in advance and to find the suitable place to paste the virtuallabel L (see FIGS. 8A and 8B).

However, the present disclosure is not limited thereto and, for example,the print data R produced in advance is not limited to one data. Inparticular, the print data R may be produced in some cases for aplurality of types of the label L such that the label L is changed insize, color, character color, etc., although the contents to be printedare completely the same. In such a case, when the desired place forpasting the label L is already determined and the operator photographsthe place with the portable terminal 10, the virtual labels LI based onthe already produced print data R are listed beside the photographedbackground image displayed on the screen of the display part 13, insteadof the color selection icons AC described above. When one label isselected from the list, the selected virtual label LI is superimposedand displayed in real time with the object image PI serving as thebackground. When another tape is selected, the selected virtual label LIis replaced. This may be repeated to allow the control circuit part 15to determine the suitability with the virtual label LI of the tape colormost suitable for pasting in the place.

Description will specifically be made of an example of determining thealready produced label L having the suitable color after the operatorfirst determines the place to paste the label L.

In this case, a place (a wall, an object, a device, etc.) to paste thelabel L is photographed by the portable terminal 10 and the photographedimage is displayed on the screen of the display part 13.

Subsequently, the operator appropriately operates the portable terminal10 to display a list of the virtual labels LI corresponding to aplurality of types of the labels L stored in advance in the storage part16 on the screen of the display part 13 (the list is displayed in aseparate frame without overlap with the object image PI or may bedisplayed in an overlapping manner).

The operator then selects a sample of one of the labels L listed on thescreen of the display part 13 and the control circuit part 15 switchesthe display of the corresponding virtual label LI to the center on thescreen. As a result, virtual overlapping display is performed in realtime as if the label L is pasted on the object image PI and thesuitability can be determined at the same time.

The arrows shown in the block diagram of FIG. 3 show an example of thesignal flow and are not intended to limit the signal flow direction.

The routines of the flowchart shown in FIGS. 4 and 9 are not intended tolimit the present disclosure to the routines described above, and theprocedures may be added/deleted or may have the order changed withoutdeparting from the spirit and the technical ideas of the disclosure. Forexample, the condition setting for the suitability determination shownat step S16 may be accepted at the same time as, or immediately after,the production/editing of the label L at step S11. In other words, thisis a routine before determining the suitability by the control circuitpart 15 and can be performed at any timing after the acquisition of theprint data R.

The techniques of the embodiment and modification examples mayappropriately be utilized in combination other than those describedabove. The portable terminal 10 is not limited to the smartphonedescribed above as long as the terminal has a camera function, a displayfunction, and a communication function and is excellent in portabilityand, for example, the portable terminal 10 may obviously be a tabletterminal as well as a digital camera, a digital video camera, etc.Furthermore, as shown in FIG. 10, the touch panel of the display part 13such as a smartphone, a digital camera, and a digital video camera canbe used as the operation part 14 and, therefore, the display part 22 andthe operation part 23 of the label producing device 20 can beeliminated.

What is claimed is:
 1. A non-transitory computer-readable recordingmedium capable of reading by a computing device, storing a virtual labeldisplay process program for execution on said computing device, thecomputing device provided in a portable terminal that is configured tooperate a label producing device configured to produce an attachablelabel and includes an operation device, an imaging device, a displaydevice, and the computing device, the program, when executed by saidcomputing device, causes the computing device to: generate virtual imagedata of said label, the virtual image data being for displaying avirtual image on said display device, the virtual image corresponding toan image of said label; output said virtual image data to said displaydevice and display on said display device a first screen, said firstscreen comprising said virtual image of said label positioned at a firstpredetermined position on said display device and a first button forinstructing said imaging device to start a continuously capturing of areal-time image; and in a case that said first button of said firstscreen is operated, acquire real image data of a desired field of viewof the real-time image captured by said imaging device and outputcomposite image data generated by combining said real image data andsaid virtual image data to said display device, and display a secondscreen that superimposes said virtual image of said label in an image ofsaid desired field of view such that said virtual image of said label isincluded at a second predetermined position on said display device,wherein said second screen includes: a first region configured todisplay said composite image data generated by combining said real imagedata and said virtual image data; a second region that is positioned atan upper portion of said first region and configured to neither displaysaid real image data nor said virtual image data; and a third regionthat is positioned at a lower portion of said first region andconfigured to neither display said real image data nor said virtualimage data and, the third region is configured to display said firstbutton, wherein on said second screen, when said portable terminal moveswhile capturing real-time image and while said virtual image of saidlabel is displayed at a predetermination position of said first region,a display is performed on said display device such that a displayposition of said virtual image of said label in said first region ofsaid display device is maintained at said predetermined position of saidfirst region regardless of variation of said field of view of thereal-time image due to the movement of said portable terminal.
 2. Therecording medium according to claim 1, wherein said label is formed bymeans of cutting a tape on which desired characters are printed in apredetermined length, said virtual image includes: a first virtual imagecorresponding to the image of said label that corresponds to said tapeto be cut; and a second virtual image corresponding to said charactersto be printed on said label, wherein said characters are to be printedon said tape; said second screen comprises at least one second buttonthat is in said first region and is capable of changing at least one ofa color of said first virtual image and a color of characters of saidsecond virtual image.
 3. The recording medium according to claim 2,wherein said at least one second button comprises a plurality of secondbuttons in said first region, and each of said plurality of secondbuttons is a button that changes a combination of said color of saidfirst virtual image and said color of characters in said second virtualimage corresponding to the color of said first virtual image.
 4. Therecording medium according to claim 1, wherein in said display device,said first predetermined position and said second predetermined positionare a same position with each other.
 5. The recording medium accordingto claim 1, wherein the program, when executed by said computing device,further causes said computing device to display a third screen on whicha third button for switching to said first screen is included and printdata is displayed in accordance with an editing operation accepted forprint contents of said label, and wherein on said first screen displayedand switched by an operation of said third button, said print data isdisplayed as said virtual image at said first predetermined position ofsaid display device.
 6. The recording medium according to claim 1,wherein on said second screen, information indicating that said imagingdevice is capturing the real-time image is displayed in said thirdregion.
 7. The recording medium according to claim 1, wherein on saidfirst screen, a first gap is provided between a first edge portion ofsaid virtual image of said label in a longitudinal direction and an endportion of said first screen, and a second gap is provided between asecond edge portion of said virtual image of said label in thelongitudinal direction and another end portion of said first screen. 8.The recording medium according to claim 1, wherein in said first regionof said second screen, a first gap is provided between a first edgeportion of said virtual image of said label in a longitudinal directionand an end portion of said second screen, and a second gap is providedbetween a second edge portion of said virtual image of said label in thelongitudinal direction and another end portion of said second screen.